Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove QueryID to fix #549 #761

Closed
wants to merge 1 commit into from

Conversation

benjimin
Copy link

If users follow the RDS documentation included in this repo, they will use this SQL yaml and encounter a cardinality explosion, because 19 separate metrics are stored for each new query that gets performed. Removed queryid to fix #549.

@sysadmind
Copy link
Contributor

@SuperQ What are your thoughts on this?

@SuperQ
Copy link
Contributor

SuperQ commented Mar 5, 2023

The query id is the primary reason that this collector exists, so, not really useful without it.

@SuperQ SuperQ closed this Mar 5, 2023
@benjimin
Copy link
Author

benjimin commented Mar 5, 2023

@SuperQ could you share a bit of explanation for why you want the query id in metrics (as opposed to logs)?

(What would the benefit be, of exporting a separate time series for each query id - given that the query id generally will never reoccur and thus the values in the time series will never fluctuate?)

This kind of flies directly in the face of Prometheus design and best practice. See, for example, the admonition from the prometheus documentation:

CAUTION:
Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

At minimum, do you think it would be prudent to update README-RDS.md to warn the user about the consequences of using the query.yaml in the manner currently advised (particularly since there are also significant financial risks noted in the #549 issue discussion)?

@SuperQ
Copy link
Contributor

SuperQ commented Mar 6, 2023

Because in many applications, the number of query ID patterns is fairly limited, and the number of database instance is also much more constrained compared to the number of application servers. This makes the total cardinality acceptable for a lot of use cases.

Knowing the about the performance characteristics of the top X queries can be extremely useful, which is why it's in the example file.

We're planning to remove support for the queries.yaml by moving the various queries into Go code. The whole feature is an anti-pattern that was added to this exporter before it was adopted by the community. As you noticed, people blindly copy the file into production.

I don't think this has anything to do with RDS or not. Any application can produce a lot of series.

What would be OK with me would be to simply comment out the whole query in the yaml file. This way it's not on by default for users that are prone to copy-pasta and don't read through the file to see what it does. I also made some suggested changes to the query but never bothered to update it. That's one of the reasons why we want to eliminate this example file. Good improvements get made locally and never make it upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pg_stat_statement metrics have unreasonable cardinality
3 participants